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Abstract 

The effects of three information distribution strategies in a course management system (CMS) on student 
performance and satisfaction were investigated. Analysis of the data found that the distribution of 
instructional materials in class resulted in significantly higher student performance on an end-of-the-semester 
technical knowledge test than when instructional materials were available to students via a CMS either at the 
beginning of the semester or one week before each of the class sessions in which the materials were to be 
covered. However, no significant differences were found among the three information distribution strategies 
on total points earned in the course or on student reactions toward the instructional experience. Possible 
explanations for these findings and implications of the findings for theory and practice as related to utilizing a 
CMS in instructional programs in higher education are provided. 
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Abstract 

The effects of three information distribution strategies in a course management system 
(CMS) on student performance and satisfaction were investigated. Analysis of the data 
found that the distribution of instructional materials in class resulted in significantly higher 
student performance on an end-of-the-semester technical knowledge test than when 
instructional materials were available to students via a CMS either at the beginning of the 
semester or one week before each of the class sessions in which the materials were to be 
covered. However, no significant differences were found among the three information 
distribution strategies on total points earned in the course or on student reactions toward 
the instructional experience. Possible explanations for these findings and implications of the 
findings for theory and practice as related to utilizing a CMS in instructional programs in 
higher education are provided. 


Introduction 

In recent years colleges and universities have made substantial investments in tools to 
facilitate the integration of technology into instruction. One of the tools being widely 
embraced by institutions of higher learning is web-based course management systems 
(CMSs). A 2003 report indicated that more than 80 percent of U.S. colleges and universities 
were utilizing CMSs (Harrington, Gordon, & Schibik, 2004). The cost of CMSs is significant, 
with many institutions spending millions of dollars on CMS development, licensing, faculty 
training, and student support services (Sausner, 2005). The University of Michigan, for 
example, recently estimated that it will spend approximately $6.8 million dollars over a 
three-year period on the development of a customized CMS. 

Surely, that level of expenditure is expected to reap rewards in terms of improving the 
quality of student learning and performance. Indeed, Sausner (2005) quotes the Vice 
President and CIO of the University of Cincinnati, Fred Siff, as stating that "the money 
[spent on CMSs] is inconsequential compared to the value and the importance because 
you're finally affecting the classroom." This statement sounds impressive, but makes one 
wonder how these web-based course management systems are affecting the classroom and 
whether those effects have a positive impact on student performance and satisfaction. 

Course management systems typically have the capacity to assist in a wide range of 
instructional tasks, including delivering course content, facilitating interaction among 
instructors and students, and evaluating learner performance (Bonk, Cummings, Hara, 
Fischler, & Lee, 1999). While CMSs have such capabilities, it has been widely reported that 
the most frequently used component of a CMS is its content presentation tools (Morgan, 
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2003; Nicoll & Laudato, 1999). Instructors typically use content presentation tools to 
distribute syllabi, class notes, participant handouts, and grades to students via the web. 

However, the practice of distributing such course information on-line may conflict with 
several principles of effective instruction. One of those principles is that for optimal levels of 
learning to occur, regardless of the type of learning outcome desired, new information 
should be arranged and presented in such a way that instruction starts with familiar 
material and builds new knowledge through a step by step progression of facts and 
connections to arrive at the desired new understandings (Gagne & Medsker, 1996; Knowles, 
1986). A concern about honoring this instructional principle in today's classrooms deals 
with the distribution of course materials via the web. For example, PowerPoint 
presentations are increasingly used in instruction, especially in post-secondary classrooms. 
When PowerPoints are made available on-line before a class session, students typically bring 
printouts of the PowerPoints to class as a guide for following the instructional process. The 
result is that it becomes more difficult for instructors to make appropriate adjustments to 
lessons without confusing and possibly irritating some students. 

A second, related principle of effective instruction that appears to conflict with the 
distribution of course materials via a CMS is that the timely presentation of new information 
during instruction helps learners focus on the learning tasks at hand and promotes their 
motivation to engage in planned learning activities (Joyce & Weil, 1996). This principle calls 
for instructors to continually check on student learning as a lesson progresses and, based 
on this on-going assessment, introduce new material at optimal moments in the lesson and 
make appropriate adjustments to the rate and way in which the new material is presented. 
However, some control over this critical instructional task is lost when course materials are 
made available on-line to learners anywhere from a day to a semester before the class 
sessions in which the materials are presented. Such advanced access to instructional 
content may result in student confusion and/or boredom and, as a consequence, may be 
counterproductive to student performance and satisfaction. 

Most empirical studies of CMSs have used survey research designs and provide little 
assistance in resolving these possible conflicts between instructional theory and practice. 
Specifically, most of the survey studies have found while students like the convenience of 
having course materials on-line (Bonk et al., 1999; Morgan, 2003), they do not believe that 
on-line access to course materials impacts their learning or performance (Young, 2004). 

Few studies, if any, have used experimental methods to examine the impact of distributing 
course materials via CMSs on academic performance and satisfaction. 

Further investigation of this topic is of great significance to undergraduate and graduate 
education in colleges and universities. Decisions regarding the delivery of instruction must 
be based not only on the technological capabilities of CMSs, but also on the impact that they 
have on student performance and satisfaction. Empirical studies that provide such 
information will provide greater insights that can be used to evaluate and possibly 
reconsider the ways that instructors use CMS tools in their courses. 


Theoretical Framework: 

Disconnects between Instructional Theory and CMS Practices 

For over 40 years, Gagne's nine events of instruction have served as a framework for 
designing effective instructional programs (Kemp, Morrison, & Ross, 1998; Smith & Ragan, 
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1999). The nine external events of instruction and the internal learning processes they 
promote (noted in parentheses) are: (1) Gaining attention (reception), (2) Informing 
learners of the objective (expectancy), (3) Stimulating recall of prior learning (retrieval to 
working memory), (4) Presenting the content (selective perception), (5) Providing learning 
guidance (semantic encoding), (6) Eliciting performance (responding), (7) Providing 
feedback (reinforcement), (8) Assessing performance (retrieval and reinforcement), and (9) 
Enhancing retention and transfer (retrieval and generalization). While instructional methods, 
such as expository and inquiry methods, sequence these instructional events in different 
orders, it is widely accepted that effective instruction does involve the use of all nine events. 

An important instructional principle underlying the effective design of these instructional 
events concerns the sequencing and organization of new information (Gagne & Medsker, 
1996; Smith & Ragan, 1999). Specifically, this instructional principle dictates the 
arrangement and presentation of new information in such a way so that instruction starts 
with familiar material and builds new knowledge through a step by step progression of facts 
and connections to ultimately arrive at new understandings. This instructional principle 
infers that the amount and complexity of instructional content should be continuously 
adjusted as a lesson progresses in response to the pace and degree of learning that is 
occurring in the classroom. This type of information presentation strategy is likely to 
promote high levels of motivation and achievement of learning outcomes. Likewise, too 
complex or too much information presented too soon may inhibit a learner's ability and/or 
desire to connect new information to current knowledge and, ultimately, diminish his/her 
ability to make meaning out of the learning experience. 

A second principle of effective instruction is that the timely presentation of new information 
is critical to the achievement of a program's learning objective (Smith & Ragan, 1999). This 
principle is anchored in the notion that the instructor or another instructional medium, such 
as a computer simulation, should decide when new information is presented and this 
decision should be made in concert with the instructional plan that has been designed to 
lead learners through the nine events of instruction (Knowles, 1986). As a consequence, 
many instructional theorists assert that when the instructor or instructional medium controls 
the time at which new information and materials are presented during a lesson, it enhances 
the focus of learners on the task at hand and promotes their motivation to engage in the 
planned learning activities (Joyce & Weil, 1996). Support for this instructional principle can 
be found in previous instructional design studies that found that providing highly structured 
and clearly ordered course materials in advance tends to foster superficial, memory-level 
encoding, but impede the far transfer of material to new learning tasks (Wilson & Cole, 
1992). A similar issue may arise when learners access course materials via CMSs. It is 
possible that such advanced access to course materials may provide so much organization 
and structure to new information that it actually impedes active processing of that new 
information and, thereby, leads to more superficial levels of student learning and lower 
levels of motivation. 

These two conflicts between instructional theory and practice drive the need to look more 
closely at the capabilities of CMSs and how they are commonly used to promote student 
learning, performance, and satisfaction in post-secondary education. Bonk et al. (1999) 
created a ten-level continuum for integrating the web into instruction in higher education in 
an attempt to clarify the pedagogical choices that instructors should consider when using 
CMSs. Levels 1 through 4 of the continuum focus on tools for information-distribution: Level 
1-Marketing the course and the course syllabus, Level 2-Student exploration of web 
resources, Level 3-Student-generated resources posted on the web, and Level 4-Course 
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resources on the web. These first four levels provide students with alternative sources for 
course information rather than provide features of the course that cannot be acquired 
elsewhere. Levels 5-10 of the continuum are distinguished from the first four levels in that 
the learning activities and tasks embedded in these higher levels contain requirements of 
the course that are not provided elsewhere. These levels include: Level 5-Repurpose web 
resources, Level 6-Substantive and graded web activities, Level 7-Course activities 
extending beyond class, Level 8-web as an alternative delivery system for resident 
students, Level 9-Entire course on the web for students located anywhere, and Level 10- 
Course fits within larger programmatic web initiative. 

Most CMSs possess the capabilities to provide instructional experiences at all ten levels of 
Bonk's continuum. However, the most commonly used component of a CMS is its content- 
presentation features, that is, Levels 1 though 4 of the web-integration continuum. For 
example, recent surveys at the University of Wisconsin (Morgan, 2003) and the University 
of Pittsburgh (Nicoll & Laudato, 1999) found that faculty used the content-presentation 
features (e.g. lecture notes, PowerPoint presentations, quizzes, and course announcements) 
much more frequently than the interactive features of CMSs (e.g. on-line bulletin boards 
and chat rooms). And, a recent nationwide survey of 4,373 students at 13 U.S. colleges 
echoes the findings from these two university surveys (Young, 2004). In spite of the 
technological capabilities of CMSs, these studies indicate that the instructional practice of 
conducting class discussions and activities in live classroom settings and limiting the use of 
CMSs to the presentation of course materials and information remains the norm. 

In large part, the rapid and widespread adoption of content-presentation features of CMSs 
in higher education results from two prevailing beliefs, one held by students and the other 
by educational administrators. From the students' perspective, a key benefit of on-line 
access to course materials is convenience (Nicoll & Laudato, 1999). Students report that on¬ 
line access allows them to download course materials if and when they choose to do so and 
this level of control enhances their ability to prepare for course sessions and organize 
course materials. 

Educational administrators have also embraced the use of content-presentation features in 
CMSs. In large part, institutional support for this feature of CMSs is predicated on the notion 
that on-line access to course materials promotes student learning and satisfaction (Nicoll & 
Laudato, 1999). A chief assumption underlying this notion appears to be that on-line access 
to course materials helps students prepare for and engage in class lectures and activities 
(Morgan, 2003). 

Interestingly, while survey research studies have found that faculty and students both 
believe that students like the convenience of having course materials on-line (Bonk et al., 
1999; Morgan, 2003), both groups also report that they do not believe that on-line access 
to course materials positively impacts student learning or performance (Harrington et al., 
2004; Young, 2004). Few studies, if any, have used experimental research designs to study 
the effects of CMSs on academic performance and satisfaction. 

As a consequence, the question of whether the distribution of course materials on-line 
enhances academic performance is presently unknown. Therefore, the purpose of this study 
was to examine the effects of information distribution strategies in a CMS on student 
performance and satisfaction in a university course. Three information distribution strategies 
were investigated: (a) on-line access to participant materials via a CMS at the beginning of 
the semester in which the materials were presented, (b) on-line access to participant 
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materials via a CMS one week prior to each class session in which the materials were 
presented, and (c) no on-line access to participant materials—all materials distributed in 
class. Two research hypotheses were investigated: (1) In-class distribution of participant 
materials will result in higher levels of student performance than either type of CMS 
information distribution strategy, and (2) In-class distribution of participant materials will 
result in greater student satisfaction with the instructional experience than either type of 
CMS information distribution strategy. 


Methods 

The population and sample, research design, outcome measures, and data analysis 
procedures are described in this section. 

Population and Sample 

The target population for this study was students using CMSs in higher education. Graduate 
students at a large mid-Atlantic university were the experimentally accessible population. 
The participating university had implemented a CMS in 2001. 

The sample was comprised of 52 students enrolled in 3 sections of a graduate-level research 
design course. This 14-week course met one evening per week and is a required course in 
three graduate programs: teaching and curriculum, health education, and training and 
development. Of the 52 students, 13 were enrolled in the Fall 2002 section, 20 were 
enrolled in the Spring 2003 section, and 19 were enrolled in the Fall 2003 section. 

As shown in Table 1, data were collected on 10 attribute variables: (a) age, 

(b) gender, (c) educational level, (d) past academic coursework in research design, 

(e) graduate program in which currently enrolled, (f) Internet usage rate, (g) relevant prior 
knowledge of concepts and principles in research design, (h) learning style, (i) cognitive 
style, and (j) self-directedness. The last three variables (learning style, cognitive style, and 
self-directed ness) were included in the study because they have been found to influence a 
person's decision about the sources and methods that they use to learn something new. 
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Table 1. Student attributes in courses where materials were distributed in class versus a CMS. 


Materials Distributed via CMS 


Attribute Variables 

Materials 
Distributed in 
Class 

At Beginning 
of Semester 

1 Week Prior to 
Each Class 



n 

% 

n 

% 

n 

% 

F 

Cognitive style 







.576 

Field dependent 

4 

30.8 

10 

50.0 

8 

42.1 


Field independent 

9 

69.2 

10 

50.0 

11 

57.9 


Educational level 







.332 

Bachelors 

13 

100 

19 

95.0 

18 

94.7 


Masters 

0 

0 

1 

5.0 

1 

5.3 


Gender 







.889 

Male 

3 

23.1 

5 

25.0 

8 

42.1 


Female 

10 

76.9 

15 

75.0 

11 

57.9 


Graduate program 







2.968 

Health Education 

2 

15.4 

5 

25.0 

0 

0 


Teaching & Curriculum 

4 

30.8 

3 

15.0 

4 

21.1 


Training & Development 

6 

46.2 

12 

60.0 

15 

78.9 


None listed 

1 

7.7 

0 

0.0 

0 

0 


Learning style 







.284 

Accommodator 

4 

30.8 

9 

45.0 

6 

31.6 


Diverger 

1 

7.7 

3 

15.0 

5 

26.3 


Assimilator 

6 

46.2 

4 

20.0 

4 

21.1 


Converger 

Past research design courses 

2 

15.4 

4 

20.0 

4 

21.1 

2.229 

Yes 

5 

38.5 

2 

10.0 

3 

15.8 


No 

8 

61.5 

18 

90.0 

16 

84.2 



M 

SD 

M 

SD 

M 

SD 


Age 

35.6 

8.60 

35.4 

8.57 

35.5 

8.74 

0.003 

Internet usage rate a 

3.6 

1.06 

4.2 

0.95 

4.3 

0.87 

1.405 

Prior research design 
knowledge b 

8.3 

1.80 

7.8 

1.88 

8.1 

2.22 

0.259 

Self-directedness c 

238.9 

26.94 

226.3 

19.28 

227.5 

27.02 

1.214 


Notes 

No significant differences among the 3 treatment groups were found on any of the 10 attribute 
variables at £ < .05. 

a Scale for Internet usage rate: 1 (Never) to 5 (More than 5 times per week) 
b Total possible points for the research design knowledge pretest = 15. 
c Total possible points for the self-directedness assessment = 290. 


Several instruments were used to collect these data. Six items were included at the end of a 
research design knowledge test to collect data on students' age, educational level, 
gender, past academic coursework in research design, graduate program in which currently 
enrolled, and Internet usage rate. 

Kolb's Learning Style Inventory was used to assess students' learning styles. This inventory 
is a statistically reliable and valid 12-item assessment tool that identifies four learning 
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styles: converger (prefers to learn by solving problems and doing concrete tasks), 
accommodator (prefers hands-on, people-oriented learning activities), diverger (prefers to 

learn by observing, brainstorming, and gathering information) and assimilator (prefers to 
learn by putting information into concise logical order) (Kolb, 1999). These four learning 
styles provide insights about one's preferences for solving problems, working in teams, and 
managing personal and professional relationships. 

The Group Embedded Figures Test (GEFT) was used to assess a cognitive style called field 
dependence/independence (Group embedded figures test, 1971). The validity and reliability 
of the GEFT have been extensively documented. The assessment activity involves the 
identification of 18 simple geometric shapes that are embedded in complex figures; more 
field independent individuals tend to be more successful at identifying the shapes embedded 
within the complex figures. Previous studies have found that field-independent learners are 
more autonomous, better problem solvers, and more likely to succeed academically than 
field-dependent learners (Pithers, 2002). 

The Self-Directed Learning Readiness Scale was used to assess self-directedness. This is a 
58-item self-report instrument, designed to measure the complex of attitudes, abilities, and 
characteristics which comprise readiness to engage in self-directed learning (Guglielmino, 
1995). This instrument has been widely used to examine the self-directedness of adults in 
professional areas such as medicine and business. Its validity and reliability have been 
extensively documented (Guglielmino, 1996). Total possible scores on the SDLRS range 
from 0 to 290. 

Research Design 

A non-equivalent control group design was used to examine the effects of three information 
distribution strategies in a CMS on student performance and satisfaction (McMillan, 2004). 
With this design, pretests on research design knowledge, cognitive style, learning style, and 
self-directedness were administered in the first class of the semester for each of the three 
treatment groups. In the ensuing 13 weeks of the semester, class notes and handouts were 
distributed to students using three different strategies. The first information distribution 
strategy involved providing students with all instructional materials in class and was 
implemented in the Fall 2002 semester. Only two features of the CMS were used with this 
class, the syllabus was posted on-line and the e-mail function was used to facilitate 
communication between the instructor and students. Because in-class distribution of course 
materials has been considered the traditional way of sharing course information, this 
treatment group was considered the control group. The second information distribution 
strategy involved posting class notes and handouts for all 13 class sessions on the CMS at 
the beginning of the semester. This strategy was implemented in the Spring 2003 semester. 
The third information distribution strategy, implemented in Fall 2003, was similar to the 
second strategy in that class notes and handouts were posted on the CMS. However, in this 
treatment group the materials were posted on the CMS only one week prior to the class 
session in which they were presented. At the end of the semester, posttests on research 
design knowledge and reactions toward the instructional experience were administered and 
the total number of points that each student earned in the course was computed. 

Participation in the study was voluntary. None of the research study assessments factored 
into the grade students earned in the course. All 52 students enrolled in the three sections 
of the course agreed to participate in the study. 


https://doi.org/i0.20429/ijsotl.2007.0i0i 11 


7 



Effects of Information Distributions Strategies on Student P erforma 


The instructional objectives, content, and activities were kept constant for all three 
treatment groups. The same instructor taught all three courses and kept a journal to record 
any questions, events, or issues that occurred in the three classes that may have related to 
the research study. 

Outcome Measures 

Two dependent variables were examined in this study: student performance and 
satisfaction with the instructional experience. 

Student performance. Two assessments were used to evaluate student performance. First, a 
15-item multiple-choice test was used to assess student knowledge of research design 
concepts and principles. The test items were selected from the instructor's manual of the 
research textbook that was used in the course (McMillan, 2004). Total possible scores on 
the test ranged from 0 to 15. A coefficient of stability was established through a test-retest 
procedure. In this procedure, 28 graduate students in 2 graduate classes during Summer 
2002 took the test two times, two weeks apart. This procedure yielded a Pearson product- 
moment correlation coefficient of .73. The internal consistency of the test was also 
calculated, with a split-half correlation analysis producing a reliability coefficient of .71. 

The second form of assessment was the total number of points earned in the course. Grades 
were based on the total number of points students earned during the semester on two 
research article critiques (110 points) and two multiple-choice exams (125 points), with the 
total possible points equaling 235. 

Student satisfaction. Satisfaction with the instructional experience was assessed with a 
written survey. The survey asked students to indicate the degree to which they agreed with 
16 statements concerning the content, design, instructor, and perceived impact of the 
course. In addition, one item asked students to indicate the degree to which the use of 
technology supported their learning in the course. The survey used 5-point Likert scales 
ranging from 1 (strongly disagree) to 5 (strongly agree). The validity and reliability of this 
survey instrument have been established in previous studies that also assessed student 
satisfaction with an instructional experience (Lohman & Finkelstein, 2000, 2002). 

Data Analysis Procedures 

A criticism of many empirical studies of on-line versus face-to-face instruction is that the 
research designs used did not investigate and control for pre-existing differences among the 
intact groups studied. As a result, the findings from many of these studies may have been 
due simply to pre-existing differences, rather than to the different instructional conditions 
being investigated (Joy & Garcia, 2000). 

In the current study, the nonequivalent control group design was used to overcome this 
limitation (McMillan, 2004). With this design, data were collected at the beginning of the 
semester on 10 attribute variables (age, gender, educational level, past research design 
coursework, knowledge of relevant research design concepts and principles, graduate 
program in which enrolled, Internet usage rate, learning style, cognitive style, and level of 
self-directed ness). These data were statistically analyzed to determine whether the three 
treatment groups differed significantly on any of these 10 attribute variables. If significant 
differences were found among the groups on any of these attribute variables, that variable 
would be included in the statistical analysis as a covariate and a multiple analysis of 
covariance would be used to test for differences among the three treatment groups on 
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student performance and satisfaction. If no significant differences were found on any of the 
attribute variables, then a multiple analysis of variance (MANOVA), using posttest scores on 


student performance and satisfaction, would be used to test for significant differences 
among the three groups. 


Results 

The statistical tests of the attribute variables and of the two main research hypotheses are 
presented in this section. An alpha level of .05 was established a priori for these statistical 
tests. 

Analysis of the Attribute Variables 

Statistical tests were conducted to examine whether there were significant differences 
among the three treatment groups on any of the 10 attribute variables. As shown in Table 
1, no significant differences were found among the three treatment groups on any of these 
variables. Therefore, it was concluded that none of the attribute variables had influenced 
the results of the study and none was included as a covariate in the tests of the research 
hypotheses. 

Tests of the Research Hypotheses 

MANOVA was used to test the effects of three information distribution strategies in a CMS 
on student performance and satisfaction. 

HI: Student performance. As shown in Table 2, the average score on the research 
knowledge posttest for students receiving course materials in class was 12.7 (SD = 0.63), 
as compared to an average of 11.1 (SD = 2.20) for students receiving course materials via 
the CMS at the beginning of the semester, and 11.3 (SD = 2.05) for those receiving course 
materials via the CMS one week prior to each class session. A MANOVA revealed a 
statistically significant difference among the treatment groups on the research design 
posttest scores, F (2, 49) = 3.218, p> = .048, with the students who received course 
materials in class scoring significantly higher than either group of students who received 
course materials via the CMS. 


Table 2. Student performance in courses where materials were distributed in class versus a CMS. 





Materials Distributed via CMS 


Student 

Materials Distributed 
in Class 

At Beginning of 
Semester 

1 Week Prior to 
Each Class 


Performance 

M 

SD 

M SD 

M SD 

F 

Knowledge Posttest 

12.7 

0.63 

11.0 2.20 

11.3 2.05 

* 3.218 

Course Points 

211.5 

16.42 

213.9 18.03 

208.6 18.91 

0.892 

* fi < .05 


In addition, the average number of total points earned by students in the course was 211.5 
( SD = 16.42) for those receiving course materials in class, 213.9 points (SD = 18.03) for 
those receiving course materials via the CMS at the beginning of the semester, and 208.6 
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points (SD = 18.91) for those receiving course materials via the CMS one week prior to 
each class session. A MANOVA revealed no significant difference among the three treatment 
groups on this variable, F (2, 49) = 0.892, 2 > -05. 

H2: Student reactions. As shown in Table 3, an analysis of the reaction survey data showed 
that students in all three treatment groups had positive reactions toward the instructional 
experience. On a scale of 1 to 5, mean scores on the 16 survey items ranged from a low of 
3.7 for students who received instructional materials via CMS at the beginning of the 
semester on their expectations for using what they learned in the course in future 
professional responsibilities (Item #15) to a high of 4.9 for both the students who received 
instructional materials in class and those who received materials via CMS at the beginning 
of the semester on their satisfaction with the clarity of the instructor's answers to their 
questions (Item #10) and overall satisfaction with the instructor (Item #11). A rating of 4.9 
was also found for the in-class materials-distribution group on their perception of the 
organization of the course content (Item #3). A MANOVA found no significant differences 
among the three treatment groups on any of the 16 survey items. 


Table 3. Student reactions in courses where materials were distributed in class versus CMS. 


Materials 
Distributed 
in Class 

Reaction Survey Items M SD 


1 . 

The objectives of this course were clear. 

4.8 

0.44 

4.5 

0.51 

4.7 

0.45 

0.999 

2. 

The content covered in the semester was 
consistent with the course objectives. 

4.8 

0.44 

4.7 

0.48 

4.6 

0.50 

0.331 

3. 

The course content was logically organized. 

4.9 

0.38 

4.8 

0.41 

4.6 

0.50 

1.382 

4. 

Large group activities and discussions were a 
constructive part of the learning experience in 
this class. 

4.6 

0.87 

4.5 

0.76 

4.4 

0.50 

0.472 

5. 

Small group activities and discussions were a 
constructive part of the learning experience in 
this class. 

4.5 

0.66 

4.5 

0.61 

4.3 

0.82 

0.521 

6. 

The course materials were useful to my learning. 

4.6 

0.51 

4.5 

0.69 

4.2 

0.83 

2.084 

7. 

The course materials were readily available. 

4.6 

0.77 

4.8 

0.41 

4.6 

0.50 

1.424 

8. 

Technology was used in a manner that supported 
my learning. 

4.2 

0.9 

4.7 

0.59 

4.3 

0.75 

2.563 

9. 

The instructor presented the course content in a 
clear and interesting way. 

4.8 

0.44 

4.8 

0.41 

4.6 

0.50 

0.574 

10. 

The instructor provided clear answers to student 
questions. 

4.9 

0.38 

4.9 

0.37 

4.6 

0.60 

1.122 

11. 

Overall, I was satisfied with this instructor. 

4.9 

0.28 

4.9 

0.31 

4.8 

0.38 

0.238 

12. 

In general, I found this class to be extremely 
challenging. 

4.6 

0.77 

4.4 

0.88 

4.3 

1.10 

0.615 

13. 

My knowledge and/or skills in research design 
increased as a result of this course. 

4.7 

0.48 

4.6 

0.5 

4.4 

0.61 

1.102 


Materials Distributed via CMS 


At Beginning 1 Week Prior 

of Semester to Each Class 


M SD M SD F 
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14. 

I expect to use what I learned in this class in 
future academic endeavors. 

4.5 

0.88 

4.1 

0.85 

4.0 

0.75 

1.297 

15. 

I expect to use what I learned in this course in 
future professional responsibilities. 

4.2 

1.17 

3.7 

0.93 

4.1 

0.78 

0.906 

16. 

Overall, I was satisfied with this course. 

4.7 

0.63 

4.6 

0.6 

4.3 

0.65 

2.219 


Note : Scores represent mean scores on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). 
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The effects of three information distribution strategies on student performance and 
satisfaction were investigated. Analysis of the data found that the distribution of course 
materials in class resulted in significantly higher student performance on an end-of-the- 
semester objective test of research design knowledge than was found when the materials 
were distributed via a course management system. However, no significant differences were 
found among the three information distribution strategies on total points earned in the 
course or on student reactions toward the instructional experience. Possible explanations for 
these findings and implications of the findings for theory and practice as related to utilizing 
a CMS in instructional programs are provided in this section. 

A possible explanation for the higher scores on the knowledge test for those that received 
course materials in class as compared to via a CMS relates to the principle of instruction 
dictating that new information should be presented in logical chunks during instruction to 
enhance learners' motivation to engage in planned learning activities and to help them focus 
on the learning tasks at hand (Joyce & Weil, 1996; Smith & Ragan, 1999). In the current 
study, when course materials were available via the CMS, students had the opportunity to 
preview new course information at any time they desired. The vast majority of students 
downloaded the PowerPoint presentations and supplemental handouts from the CMS in 
preparation for class. Many had read the new material prior to class and some had 
prepared questions about the material for class discussion. In contrast, when course 
materials were distributed in class, the instructor gradually introduced new material, 
adjusting the pace of information distribution in accordance with the rate and degree of 
learning that was occurring. Additionally, as new material was distributed in class the 
instructor explained its relevance to the learning task at hand. The continuous adjustment 
of the pace at which new material was introduced and the just-in-time explanation of the 
relevance of the material to the learning task may have enhanced students' levels of 
motivation and ability to focus on key aspects of the lessons and, ultimately, resulted in 
greater acquisition of research design knowledge than was the case for students who 
acquired course materials via the CMS prior to class. 

Yet, no significant differences were found among the three treatment groups on the second 
measure of student performance, total course points. A plausible explanation for this finding 
was that the evaluative activities for which course points were earned required higher levels 
of thinking than the research design knowledge test did (Kemp et al., 1998). Students 
earned course points for their performance on two multiple-choice exams and two research 
article critiques. The second exam and both research article critiques required students to 
make critical judgments about the research designs and procedures that had been used in 
published research studies. These higher-level cognitive activities may have enabled 
students, regardless of which treatment group they were in, with a better opportunity to 
demonstrate their learning than was the case with the knowledge test. 

A puzzling finding in the current study was that there were no significant differences among 
the treatment groups on student satisfaction with the instructional experience. It has been 
widely reported that the vast majority of students believe that accessing course materials 
on-line is convenient and helps them organize and prepare for class. However, students who 
accessed course materials via the CMS in the current study did not perceive that the use of 
technology or the availability of course materials was substantially better than students who 
received materials in class. Interestingly, several students in the in-class treatment group 


Effects of Information Distribution Strategies 
12 


Discussion 


https://doi.0rg/1 0 . 20429 /ijsotl.200 7.010111 


12 



IJ-SoTL, Vol. l [2007], No. 1, Art. 11 


International Journal for the Effects of Information Distribution Strategies 

Scholarship of Teaching and Learning ^ 3 

http://www.aeorgiasouthern.edu/iisotl 

Vol. 1, No. 1 (January 2007) 

ISSN 1931-4744©Georgia Southern University 


indicated at the beginning of the semester that they wanted course materials (e.g. class 
notes and PowerPoint presentations) to be posted on-line. When the instructor explained 

that the CMS would not be used for that purpose, students seemed mildly disappointed. 
However, that disappointment dissipated within a few weeks and at the end of the semester 
the ratings of the in-class treatment group on the quality of the instructional experience, 
including the utilization of technology in instruction, were similar to both CMS groups. 

When considering the implications of this study's findings, several of its limitations should 
be noted. The treatment was administered in three sections of the same course over a 
three-semester period. This design enabled the use of one instructor for all three treatment 
groups, thereby minimizing the likelihood that different teaching styles or teaching ability 
levels influenced the results of the study. However, two inherent limitations of the research 
design required actions to be taken by the researcher to enhance the confidence that the 
results were due to the treatment and not to something else. The first limitation was that 
three intact student groups participated in the study. To examine whether this limitation 
impacted the study's results, a statistical analysis of 10 attribute variables was conducted. 
No significant differences among the three groups were found on any of the 10 variables. A 
second limitation was that the three student groups experienced the treatment at different 
points in time. As a consequence, it was possible that events may have occurred in one of 
the semesters that influenced the performance and/or satisfaction of students in that group. 
A journal was kept by the instructor to record any such events in class sessions, on campus, 
or in the news that related to the integration of technology in the instructional process. Only 
one event was observed during the administration of the treatment that pertained to the 
use of CMS. Because inclement weather caused the last class session of the in-class 
treatment group to be cancelled, a research article that was to be used for the final exam 
was distributed to students via the CMS rather than in class. Despite these limitations, the 
findings of this study have important implications for theory, practice, and the future 
research of CMSs in relation to promoting student performance and satisfaction in higher 
education courses. 

Theoretical Implications for Integrating CMSs into Instruction 

The current study's findings provide empirical evidence to extend understandings of Bonk's 
ten-level web-integration continuum (Bonk et al., 1999). Specifically, this study's findings 
suggest that higher levels of student performance and satisfaction are not likely to occur if 
instructors utilize CMS tools that pertain only to the first four levels of the continuum; that 
is, those tools that enable learners to use the web as an alternative source for acquiring 
course-related materials and information. This finding is particularly concerning because 
most instructors in post-secondary education have NOT extended their use of CMSs beyond 
the first four levels of the web-integration continuum (Bonk et al., 1999; Young, 2004). 

As a consequence, it is reasonable to suggest that the upper six levels of the web- 
integration continuum is where the most promise and opportunity resides for utilizing a CMS 
to enhance student performance and satisfaction (Bonk et al., 1999). Levels 5-10 of the 
continuum are distinguished from the first four levels in that the learning activities and 
tasks in the upper levels of the continuum contain requirements of a course that are not 
provided elsewhere. These levels range from Level 5, where group discussions and problem¬ 
solving activities are facilitated, to Level 10, where the course fits within larger 
programmatic web initiatives. Clearly, this study's findings show that the conventional 
practice of using a CMS as an alternative source for course materials and information 
provided in class does not lead to superior learning or affective outcomes. Yet, the question 
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remains as to the potential benefits to be derived from using a CMS to accomplish learning 
tasks and activities that go beyond those conducted in face-to-face classroom settings. 
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Implications for Integrating CMSs into Instruction 

This study's findings give rise to two important implications for integrating CMSs into 
instruction. First, the study's findings provide little evidence to warrant continuing the 
conventional practice of using a CMS as a alternative way of presenting course materials 
and information. CMSs cost millions of dollars to purchase and maintain. Instructors must 
spend countless hours learning how to use the technology, preparing materials for CMSs, 
and monitoring student use of course websites. And, for what benefit? No evidence was 
found in this study to indicate that on-line access to course materials yields any positive 
dividend in terms of student learning or satisfaction. 

A second implication of the study's findings is that current instructional practice is 
underutilizing the technological and pedagogical capabilities of CMSs. As such, instructional 
practice must move beyond using CMSs to replicate learning activities and materials that 
are already provided in class. Instead, CMSs should be used to conduct learning activities, 
such as facilitated group discussions and problem-based learning experiences, that 
supplement in-class activities as well as to provide additional learning resources that are not 
available in class. In other words, pedagogical practice must move into the upper six levels 
of the web-based continuum for integrating technology into instruction (Bonk et al., 1999). 

However, instructors cannot be expected to move higher in the technology-integration 
continuum without additional time, training, and technical assistance. Instructors'teaching 
loads would need to be temporarily reduced to provide time to enhance their pedagogical 
skills in designing, implementing, and evaluating technology-mediated learning activities, 
such as electronic conferencing and journaling, on-line group debates, and web-based team 
projects. Instructors would also need additional time to develop the technical skills required 
to use the more advanced and complex features of CMSs. In addition, support staff with 
expertise in designing instruction for web-based environments would be needed to assist 
instructors with the pedagogical and technical concerns that are certain to arise as 
instructors work toward integrating more web-based learning events and activities into their 
instructional programs. 

Implications for Future Research of CMSs 

Further research in two areas would help clarify the potential value of CMSs in higher 
education. First, because the current study was conducted with graduate students, 
replications of the study with undergraduate students would provide a greater 
understanding of its generalizability to student populations in higher education settings. 
Second, future research studies must investigate the effects of integrating CMSs at the 
higher levels of the web-integration continuum on student performance and satisfaction. 

This is a crucial area of research because, at the present time, there is little empirical 
justification to advocate continuing the conventional use of course management systems as 
an alternative means of presenting course information and materials. 
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